Yes, this is definitely needed and I'd schedule implementation right after API spec v0.3 as the API spec v0.0.2 is not formalized enough to be really testable.

We need to think about tooling for both the JSON responses and for the processed EO data itself.

JSON responses can probably tested relatively easy with tools

• testing against the OpenAPI spec such as https://assertible.com/ - unfortunately, it is not open-source and I couldn't find anything that is open-source and supports OpenAPI 3.0. Their free plan might be too limiting.
• testing against yet to be made JSON schemas.

For EO data validation we may need a custom implementation or does anybody know of any EO comparison / test suite? File comparison on a binary or checksum level probably won't give good results as files will probably always slightly change.

We also need to test the "work-flow" / complete use cases. That probably needs to be custom made by us.

We might want to have both smaller data sets to test against and can be run on a single machine in some minutes, but also larger ones for more extensive tests. We still need test data. Can pilot users provide any data?

For completeness:

Edzer asked on the mailing list:

Is anyone aware of existing reference test datasets that have been used and published for EO processing purposes?

Response from Noel:

I can think of a number of datasets that people test with, but in general the source data changes often enough that I don't know of anyone that's maintaining "reference" versions. For instance, the USGS developed a set of images for use in the ACCA and FMASK cloud masking experiments, but that dataset hasn't been updated to keep up with Collection 1 activities. Same with e.g.: MODIS Landcover.

But we treat a number of datasets as "canonical" as they're fairly ubiquitous and are freely available with permissive licenses:
Elevation - USGS/SRTMGL1_V003
Landcover - MODIS/MCD12Q1
Crop Type - USGS/NLCD
Nighttime lights - NOAA/DMSP-OLS/CALIBRATED_LIGHTS_V4
High res optical - USDA/NAIP
International Borders - USDOS/LSIB (GAUL is the standard, but not permissive).

I signed up for assertible.com and had a look at it. It offers quite a wide range of assertion types and the free plan does allow a fair amount of tests. They offer the possibility to import from a Swagger/OpenAPI (v2 or v3) definition (among others, like a Postman collection or "API Blueprint").

So I had a try importing our swagger.json into their system. The import worked, but didn't consider all the available information. For example, for GET /capabilities/output_formats the import created only one assertion: whether it returns 200 OK. And that despite the fact that the swagger.json contains a schema definition for the answer expected from that endpoint.

Assertible.com therefore doesn't look like THE tool that could automatically turn our OpenAPI definition into a testing suite. Manual conversion of the schemas would be necessary. If that's done, the service could be quite powerful.

My colleague Marc Jansen (https://github.com/marcjansen) suggests to look at https://www.soapui.org

Thanks for the idea @neteler. I installed it and my impression isn't positive. I wasn't able to find an import option for Swagger, which immediately makes this suite a worse choice than Assertible. There's supposed to be a plugin that adds Swagger import, but it appears to be broken and unsupported.

Also, looking at their releases section, SoapUI seems to be a stale project in general. I couldn't find hints on whether it even supports OpenAPI v3, and therefore doubt it does.

To me it appears that while the SoapUI software (or the Ready API suite, of which it is a part) is still sold (the SoapUI pro version is ca. 600 euros per year), SmartBear's focus now lies on Swagger Editor and Swagger UI etc. as indicated e.g. by this blogpost, which mentions these two tools but not "the old ones" - I guess they're not being developed anymore.

Further input is highly appreciated!

... ok, we didn't check SoapUI in detail. Will keep eyes open!

I found another tools we should check out later as OAS3 support is still work in progress: https://bitbucket.org/atlassian/swagger-request-validator

During EODC Forum it was also asked by users to bring this forward with regards to comparability of results.

Comment from @TomMiksa:

The first point in the list - about comparability - is very important. It actually expresses the need for reproducibility - what happens when I run the same process at different back ends? Can I pick the back ends at random? Are the cost criteria the only reason to switch? If you don't like the term reproducibility, we can also call it non-functional requirements or quality of service (QoS).

When using a common API, I would expect the services to have the same quality of service, or if they differ, they should be explicitly documented.

To address this, we could introduce into our API a function like "describeYourself" in which the back end could provide all necessary information about its QoS. What exact information it should be, is something we can discuss.

Our current architecture assumes we don't have any central service that routes requests to back ends - everything is decentralized. Adding a function like "describeYourself" would not change this architecture. Furthermore, if in the future, one would like to present an overview/comparison in the official webpage, then this could be done by making a call to each back end and simply aggregating the results.

That's interesting: opengeospatial/ogcapi-features#129
It may allow us to just specify JSON schemas for our API in the API spec and then use JSON schema validators for validation, which is not yet a thing for OpenAPI schemas.

Work will start here after releasing API version 0.3: https://github.com/Open-EO/openeo-backend-validator